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Abstract 

Background: In many academic settings teaching a particular topic is applied to every student enrolled in the 
same academic year, it is a difficult task for researchers to design a randomized control group study. This research 
aimed to estimate the effect of teaching management and planning on increasing academic planning behavior 
(APB), using propensity score matching (PSM). 

Methods: In a cross-sectional survey utilizing a self-reported structured questionnaire on a systematic random 
sample of 421 students in Hanoi Medical University, one of the eight medical schools in Vietnam, this evaluation 
study adopted regression procedures to assess model fit, then PSM to create a matched control group in order to 
allow for evaluating the effect of management education. 

Results: The study showed both direct and indirect effects of the education on behavior. After PSM to adjust for 
the possible confounders to balance statistically two groups - with and without management education, there is 
statistically a significant difference in APB between these two groups, making a net difference of 18.60% (p < .05). 
The estimated 18.6 percentage point increase can be translated into the practice of APB by 670 students in the 
population. This number of academic planners can be attributed to a high recall of important management and 
planning education. 

Conclusions: The study provided theoretical as well as practical implications to guide the design of the education 
and evaluation of teaching. 



Background 

Academic planning behavior (APB) may involve differ- 
ent tasks and skills of students - setting goals, planning 
activities, considering alternatives, monitoring and 
reflecting, as well as readjusting plans to meet progress 
rates [1,2]. A behavior as such plays a critical role in 
improving students' academic performance [1-4]. How- 
ever, in various higher education settings APB has not 
been widely practiced among students; consequently, 
their academic achievement has been limited [5]. Teach- 
ing a topic on management in order to prepare or to 
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direct students toward a well-planned study is therefore 
important if higher education programs are to be 
effective. 

A theoretical framework for this study is based on a 
comprehensive conceptual model by Kincaid [6,7] (Fig- 
ure 1) that has been adapted from a wide range of lit- 
erature resources. Under this theory, the psychological 
influences including knowledge, attitude, social norm, 
intention, self-efficacy, and others can be combined as 
ideation. Specific communication interventions may be 
designed to influence only one or several types of psy- 
chological processes. All sorts of psychological processes 
are expected to affect behavior even if communication is 
designed to influence only one of them. Communication 
affects behavior indirectly by providing information that 
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Figure 1 A Model of Strategic Communication and Behavior Change. 
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changes one or all of such processes. Exogenous deter- 
minants including demographic, socioeconomic and 
contextual characteristics affect endogenous variables - 
such as the exposure to the communication, ideation 
and behavior. 

To review a higher education program, teachers and 
researchers would like to be able to estimate the effect of 
teaching a topic designed to change a behavior or to 
enhance academic performance. It has been well recog- 
nized among researchers and evaluators that calculating 
effectiveness is the most important part, but is sometimes 
the most challenging [8]. One can not claim a particular 
amount of behavior change without a causal attribution. 
A causal inference must be reached that attributes the 
net change in behavior to exposure to the intervention 
and not to other impacts or, worse yet, to changes that 
occurred before the intervention was implemented. 

Because in many academic settings teaching a particu- 
lar topic is applied to every student enrolled in the same 
academic year, it is a difficult task for researchers to 
design a randomized control group study. Many educa- 
tion and communication scholars have concluded there- 
fore that it is virtually impossible to draw such a causal 
inference [9]. Without such a capability, how can we 
ever justify the influence of an education program? 

In Vietnam, for medical students, to graduate as a medi- 
cal doctor bachelor every student has to successfully pass, 
across six academic years - from 1 to 6, all topics, one of 
which is management and health management science 
which is offered by the Department of Health Organiza- 
tion and Management at the fifth academic year, usually 
during the annual fall (from August to October). Although 
several studies have been conducted, most looked at stu- 
dents' academic scores as main indicators to measure the 
effectiveness of teaching, while intermediate behaviors like 



planning skills which help students achieve higher aca- 
demic progress, have remained under-researched. APB has 
become increasingly important as students were claimed 
under pressure in terms of time management due to a 
variety of theoretical and practical topics enrolled simulta- 
neously in the same year. It is further pressure placed on 
students in terms of academic achievement due to a great 
volume of knowledge and skills to be learned within uni- 
versities. To assess the impact of an education program, 
researchers and evaluators may adopt different methods. 
Among these, PSM is highly recommended as it is one of 
the strong statistical techniques [6-8,10,11] as to be dis- 
cussed. This method can help to reduce selection bias as it 
allows for quasi-experimental contrasts between students 
in receiving "treatment" and "control" groups based on 
their observed characteristics. Proper use of PSM should 
also allow for rigorously derived and relatively unbiased 
estimates of communication's or education's effects on 
participants' behavior [12]. Because of its ability to reduce 
selection bias, PSM has become increasingly used in the 
fields of education [13,14], communication [6-8,10], medi- 
cine and epidemiology [15], policy evaluation [16], eco- 
nomics [17], psychology [18]. Although commonly applied 
in diverse disciplines in many contexts, too little has been 
achieved in measuring APB as an intermediate outcome 
that helps to enhance students' academic progress. 

The purpose of this study is to use propensity score 
matching (PSM) to estimate the impact of teaching man- 
agement on APB among medical students in a Vietna- 
mese medical education context. 

Methods 

Study design and setting 

An evaluation study was conducted with a cross-sec- 
tional sample survey using a self-administered 
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structured questionnaire. The study was carried out in 
Hanoi Medical University (HMU), located in northern 
Vietnam. This is one of the eight medical universities in 
the country. Similar to all eight medical universities, 
HMU offers teaching a managerial topic with the same 
contents, one of which is management and planning 
science designed to help students develop planning 
knowledge and skills, which is helpful to their current 
study and their future health care career. 

Participants and sampling procedure 

A representative sample survey of 421 students across 
academic years 1-6 was conducted within November to 
December 2009. This sample was obtained based on a 
systematic random technique for the survey given a 
sampling frame of 3145 students, a total population size 
of undergraduate students from the first to sixth year. 
The sampling frame was provided by the Director of the 
HMU Undergraduate Training Department. Following a 
random start, 421 students were selected, using a fixed 
interval of 7. 

Data collection 

The material for data collection was a self- administered 
structured questionnaire. It included 4 sections, first 
asked students about their individual and social charac- 
teristics, second the level of access to the topic of man- 
agement science, third knowledge, attitudes and self- 
efficacy of academic planning and finally APB during 
the past and current year. 

As a procedure, the study was first approved by the 
HMU Undergraduate Training Department, then all 
selected students were informed of the study objectives 
and contents. 95% of the sample gave their informed 
consent to participate; because 5% refused the study, we 
used the same sampling strategy to approach more stu- 
dents until a full size of 421 students was reached. Both 
male and female medical students were surveyed, using 
the anonymous self-reported manner. Each student 
completed a confidential questionnaire for 15 to 20 
minutes either before or after a lecture he or she 
attended as well as after permissions received from tea- 
chers. The Institutional Review Board (IRB) of the 
HMU has not required a formal ethical approval for the 
survey. 

This study was informed by a pilot survey to validate 
the instrument. The pilot showed that the instrument 
was technically feasible for the main survey. 

Measurement 

Exposure to management education can be measured as 
a state whether a student has learnt management topic. 
However, a good method of communication or educa- 
tion evaluation requires a valid, reliable measure of 



recall of key contents at the interval level of measure- 
ment. In this study, therefore, an open-ended question 
was added to ask students to remember and list all pos- 
sible contents of lessons they have learned. To transfer 
textual data into standard contents, a principal investi- 
gator from the topic of management and health man- 
agement, who knew well about the topic and its 
contents, closely looked at all textual responses, then 
classified if each of them would go under the standard 
contents of the topic. In total 8 standard contents were 
identified. The above question formed a continuous 
scale measuring the level of recall that ranged from 0 to 
8 contents with a median of 4. Recall of at least one 
content was 90%. To simplify this measurement and 
accommodate the logistic regression and propensity 
score analysis, the scale was classified into 1 and 0 
(higher versus lower recall). 

Management and planning knowledge was measured 
by 8 true/false/don't know items such as "a good aca- 
demic planning includes setting goals and considers 
timeline frame and resources". Scoring the knowledge 
scale was accomplished by dichotomizing each item into 
a value of 1 (correct) and 0 (incorrect or don't know) 
and then summing the item values to form a composite 
score with higher scores reflecting better knowledge 
(Cronbach's alpha = .50; mean score = 6.32, SD = 1.96). 
Attitude toward academic planning was measured with 
seven 5-point semantic scale {bad-good) from 1 (nega- 
tive evaluation) to 5 (positive evaluation) such as "how 
good or bad would it be if you talked about academic 
planning for each topic with your friends every year?' A 
composite score was obtained by summing responses to 
items with higher composite scores indicating higher 
levels of attitudes (Cronbach's alpha = .55; mean score = 
16.26, SD = 3.76). Social norm toward academic plan- 
ning was assessed with seven 5-point semantic scale 
[untrue-true) from 1 (negative evaluation) to 5 (positive 
evaluation) such as "Most people - family, teachers and 
close friends - who are important to you think you 
should talk about academic planning with your friends 
every year 7 " A composite score was obtained by sum- 
ming responses to items with higher composite scores 
indicating higher level of social norm (Cronbach's alpha 
= .85; mean score = 13.57, SD = 4.30). Intentions for 
academic planning are measured by asking students to 
rate on a 5-point semantic scale ranging from very unli- 
kely (1) to very likely (5) such as "during the next few 
months, you intend to talk about academic planning 
with your friends for each topic?". A composite score 
was formed by summing responses to items with higher 
scores indicating higher levels of intentions (Cronbach's 
alpha = .83; mean score = 15.62, SD = 5.17). Self-effi- 
cacy of academic planning was assessed with seven 
items tapping perceived difficulty of academic planning 
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on a 5-point semantic scale from very hard (1) to very 
easy (5) such as "how hard would it be for you to advice 
or persuade your friends to make academic planning?' A 
composite score was obtained by summing responses to 
items with higher scores reflecting higher levels of self- 
efficacy (Cronbach's alpha = .63; mean score = 18.61, 
SD = 5.10). These five related sub-constructs represent- 
ing the cognitive and social interaction component of 
ideation were used to construct the measures of idea- 
tion. For the logistic regression analysis, using a cut-off 
of 50% the measures were split into 1 and 0, corre- 
sponding to higher and lower levels of ideation. 

Past year and current APB were combined to con- 
struct a continuum of behavior with the following five 
scale values: 0 = never, 1 = rarely, 2 = occasionally, 3 = 
usually, and 4 = always (Cronbach's alpha = 0.70). 
Combining these two items into a single outcome vari- 
able has two advantages in that it makes the measure- 
ment more valid and reliable as well as allows the 
analysis of the impact of education on students' APB. 
The levels of APB met the minimal requirements of 
order with respect to ideation with the support from 
[19] one-sided test of significance performed with p = 
.01. This level of probability indicated the rejection of 
the null hypothesis of equality of levels and supporting 
the alternative hypothesis of order. To make logistic 
regression analysis and estimation of the impact possi- 
ble, the scale of the single APB was classified as 1 and 0 
reflecting more and less frequent level of APB. 

Socioeconomic status was measured by means of liv- 
ing total money received from family categorized into 
two levels - higher and lower socioeconomic status. Age 
was measured by number of years categorized into 
higher and lower groups. Level of university education 
was measured by ordinal number of academic years 
categorized into seniority (three final academic years) 
and juniority (three first academic years). Origin of per- 
manent residence was classified into urban and rural 
area. Demographic characteristics including gender, eth- 
nics and religion were also included. Further, prior 
research and literature has indicated that a student's 
father and/or mother occupation - called parental occu- 
pation (white vs. blue collar) predicts his or her study 
behaviors and academic achievement [20-22]. To maxi- 
mally reduce the potential for selection bias, we 
included these covariates in the model predicting pro- 
pensity to receive education [23]. 

Statistical analysis 

The software program "STATA version 10.0" was 
applied to all the following statistical procedures. 
Simple proportion differences 

We used Chi-square tests to determine whether propor- 
tion differences in APB of students receiving and not 



receiving management and planning education were sta- 
tistically significant. We used a p value of .05 for these 
analyses. We considered the results obtained from Chi- 
square tests of these proportion differences as a "bench- 
mark" for the results obtained from analyses using PSM 
as referred to below. 
Logistic regression modeling 

The model of direct and indirect effects used for this 
analysis requires three equations, one for each endogen- 
ous variable: APB, ideation, and recall of the planning 
contents educated. Each of such equations is presented 
as follows: 

Yii = Po + PiY 2i + P 2 Y 3i + P 3 Xi + |Jbi 
Y 2 i = Yo + YiY 3 i + Y 2 Wi + Vi 

Y 3i = a 0 + CXiZi + 

Where Y u is APB for subject i, Y 2 i is ideation, Y 3i 
represents exposure to management and planning edu- 
cation for subject i, X b Wi and Zi are matrices of exo- 
genous socioeconomic and demographic control 
variables, the three p, two y and one a coefficients are 
parameters to be estimated from the data, and vi, 
and t]i are the disturbance (residual) terms. Because 
APB, ideation, and exposure are measured with a bin- 
ary scale, logistic regression is used to estimate the 
parameters of the equation. The differentiation among 
the X, Z, and W matrix of exogenous control variables 
indicates that each endogenous variable should be 
determined by exogenous variables not included in the 
other two equations. However, some overlap of exo- 
genous variables can be acceptable, but each endogen- 
ous variable must have at least one exogenous control 
variable that is excluded from the equations for all 
other endogenous variables [24]. The arrows indicate 
the hypothesized direction of influence and effects, 
that the three error terms are uncorrelated, and that 
there is no third unobserved variable that accounts for 
any of the hypothesized relationships. In examining the 
model fit of these three equations, tests for endogene- 
ity are used to determine if there are any unobserved 
variables responsible for the observed relationship 
between exposure and the outcome, and if the rela- 
tionship is reciprocal. The thread of endogeneity can 
be ruled out if the disturbance terms, s2 or vi and s3 
or §i, are statistically uncorrelated as reflected as rho 
and if they are not statistically significantly when 
added to the equations for behavior ad ideation, 
respectively. If the coefficient of the estimated error 
term (s2, s3) is not significantly different from 0, one 
would accept the null hypothesis that the suspected 
endogenous variable is an exogenous variable, and that 
there is no unobserved variable affecting on the out- 
come y3 for the treatment (education exposure); and, 
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therefore, y3 can be regarded as exogenous, and simple 

regression can be adopted [8]. 

Creating the matched sample using propensity score 

matching 

A propensity score is the probability of being exposed to 
a treatment or an intervention given a set of observed 
covariates, X. The method as this was developed as a 
means to balance the treatment and control units so 
that a direct comparison would make a valid conclusion. 
The technique was found robust as recently in practice 
it was difficult, if not possible, to match on more than 2 
variables. PSM therefore makes it feasible to create an 
unexposed comparison group that is statistically equiva- 
lent on the average to subjects who are exposed to the 
treatment [25,26]. For research survey, a single score for 
matching is generated using statistically regressing expo- 
sure on all of the variables that determine exposure and 
also may be related to the outcome variable [8]. 

This technique requires a two-stage process. Stage 1 
involves the use of a logistic or probit regression model 
to calculate all respondents' propensity for experiencing 
a treatment of interest, in this case, receiving manage- 
ment and planning education (MPE). The propensity 
score is defined as follows [25]: 

p(T) = Pr2 = E{T | S} 

Where p(T) is the propensity to be placed into MPE, 
T indicates that a student did or did not receive MPE, 
and S is a vector of covariates influencing whether the 
student did or did not receive MPE. In Stage 2, we used 
the estimated propensity scores obtained in Stage 1 to 
match students who did and did not receive MPE. To 
obtain a full sample, we used stratification matching 
which uses all treatment and control cases. Using the 
STATA 10.0 "atts" command, the full range of sample 
members' propensity scores is divided into propensity 
score strata, or blocks, each of which includes treatment 
and control cases with the same or nearly the same pro- 
pensities for receiving the treatment. The number of 
appropriate strata depends on the number necessary to 
gain a balanced propensity score. Within each of these 
strata, the ATT is calculated, and then the ATT's across 
strata are averaged to produce a final ATT. 

Results 

Sample characteristics 

In total, 421 undergraduate students were asked to par- 
ticipate in the study. The proportions were fairly evenly 
distributed by gender (male vs. female, each with -50%), 
academic years (from year 1 to year 6, each -16.5%), 
and place of permanent residence (urban vs. rural, each 
~ 50%). The mean age of all participants was 21 years 
(range 18-27 years: SD = 1.89). The distribution of 



monthly stipend from family for living and studying was 
VND1.36 million (range .3-10: SD = .72; US$1 = 
VND19,000). The main differences were in ethnicity 
(most were the Kinh, a major ethnic group in Vietnam), 
religion (most were the followers of Buddhism, ancestor 
worship and non-religious), and parental occupation, 
with white collars accounting for almost 70% of the 
total. 

Simple proportion differences 

Table 1 compares APB proportions between the study's 
subsamples of those who exposed to and did not expose 
to MPE. There were unadjusted proportion differences 
in all categories of APB between these two groups. 
Results indicate that students receiving MPE displayed 
higher proportions of APB. Overall, students who had a 
higher recall of MPE contents showed a higher propor- 
tion of APB compared to those with a lower recall 
(71.51% vs. 42.28%; with a relative gain of 29.23 percen- 
tage points). This difference is statistically significant at 
the p < .001 level. 

Logistic regression modeling for predictors of recall, 
ideation and APB 

The results of the logistic regression analysis using pro- 
bit procedure are shown in model 1 of Table 2. Descrip- 
tive statistics for each variable are reported in the 
second column. The equation for recall, ideation, and 
APB were identified with 9 exogenous variables. Results 
of the exclusion tests are reported at the bottom of the 
column for each equation. No statistically significant 
effects of exclusion were found. The proportion of the 
variance explained by the equation for recall, ideation, 
and APB was .27, .22, and .32, respectively. As expected, 
the best predictor of APB were ideation toward APB 
(.75, p < .001). Recall of MPE contents also had a direct, 
significant effect on APB (.57, p < .05) as well as an 
indirect effect through its effect on ideation (.50, < .05). 
These results indicate that ideation is a mediating (inter- 
vening) factor between MPE recall and APB and hence 
one of the causal pathways by which education has an 
effect. This indirect effect is what the content of the 
education was promoted to achieve, which confirms the 
theory as well as the empirical impact. The direct effect 
of MPE recall on ideation was statistically significant 
after controlling for the effect of a number of demo- 
graphic and socioeconomic variables. 

The effects of the exogenous control variables are also 
theoretically relevant. Recall of the MPE was related to 
age, gender and place of permanent residence. Ideation 
about APB was related to age and parental occupation. 
APB was related to academic year. There is no reason 
why religion would be related to recall and APB, so it 
was excluded from those equations. However, religion 
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Table 1 Dependent variable proportions for MPE students and non-MPE students 



Dependent Variable 
(APB) 



Exposed Students 
n(%) 



Non-Exposed Students 
n(%) 



(X 2 



APB during the past year 
APB during the current 
year 



APB during the past year 
APB during the current 
year 



Enrolled in MPE (n = 146) 

65(44.52) 
69(47.26) 



Non-Enrolled in MPE (n = 275) 

85(30.91) 
99(36.00) 



Exposed to 50% or more of all MPE sessions (n = Exposed to just less than 50% of all MPE sessions (n 
123) 298) 



57(46.34) 
61(49.59) 

Higher recall (n = 123) 



93(31.21) 
107(35.91) 

Lower recall (n = 298) 



Composite APB 



88(71.51) 



126(42.28) 



*p < .05; **p < .01: ***p < .001 



was related to ideation (including attitudes, social 
norms, and intentions), albeit not significant. Because 
age, gender, parental occupation, and place of perma- 
nent residence may be neither as well as would nor 
expected to be, related to APB, so they were excluded 
from the APB. Theoretically, it was found that place of 
permanent residence would not influence ideation, 
thereby being excluded. 

Propensity score analysis 

The results underlying the propensity score analysis are 
reported in the second model presented in Table 2. Pro- 
pensity score analysis was done for the level of recall 



(high versus low/no recall). To test for endogeneity and 
support for the assumption of strong ignorability, we 
treated the problem as a two-equation system. The 
results showed that high recall of MPE contents had a 
statistically significant direct effect on APB (.47, p < .05) 
after controlling for the remaining variables in the 
model. Biprobit analysis of these two equations was con- 
ducted to test for the endogeneity of level of recall in 
the APB equation. The correlation of the error terms 
was not statistically significant (rho = .07, p > .05), indi- 
cating that MPE recall may be considered as exogenous 
in the equation of APB. When the error term from the 
equation for recall was added to this equation, it was 



Table 2 Results of the structural equation model and propensity score analysis of APB 



Variables Description % or mean Model 1 Model 2 

(range) Logistic regression (probit procedure) Propensity Analysis 







Recall 
(Probit 
coef.) 


Ideation 
(Probit 
coef.) 


APB 
(Probit 
coef.) 


Recall 
(Probit 
coef.) 


APB 
(Probit 
coef.) 


Age (older vs. younger) 


21.04 
(18-27) 


1 .22*** 


0.20*** 




1.22*** 




Gender (female vs. male) 


53.00 


0.82*** 


0.17 




0.80*** 




Parental occupation (white vs. blue collar) 


41.09 


0.07 


0.29* 




0.06 




Place of permanent residence (urban vs. 
rural) 


47.27 


0.19* 






0.18 




Ideation (higher vs. lower) 


18.05 






0.75*** 




0.75* 


Recall (higher vs. lower) 


29.22 




0.50* 


0.57* 




0.47* 


Academic year (senior vs. junior) 


51.54 






0.19* 




0.10 


Living stipend level (million Vietnamese 
Dong) 


1.36 
(0.3-10) 


0.14 




0.11 


0.15 


0.11 


Religion (yes vs. no) 


2.14 




0.60 








Number of cases 


421 


421 


421 


421 


421 


421 


Variance explained (adjusted R 2 ) 




0.27 


0.22 


0.32 


.20 


0.12 


Exclusion test (model fitness) (p) 1 




NS 


NS 


NS 


NS 


NS 



Test for endogeneity - Biprobit rho for Model 2: 0.07 (-0.3 0-0.43) 
rho-based p: Chi 2 = 0.1378; p > 0.710 

Statistical significance: *p < .05; **p < .01; ***p < .001; NS = not significant 
Exclusion of a variable for model identification is indicated by (-). 
''The likelihood ratio test was used for binary dependent variables. 
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not statistically significant as well. This suggests that 
ordinary probit regression may be used to estimate the 
direct effects of high recall of MPE contents and that 
the potential effects of omitted or unobserved variables 
may be ignored. The result of the propensity score ana- 
lysis as informed from these equations is therefore 
expected to approximate what we would expect from a 
randomized control group design. 

Table 3 shows the results of the stratification of pro- 
pensity score. We used stratum-based PSM in order to 
maintain the full sample design and yield results based 
on the same cases as the SEM analysis. The propensity 
score - probability of high recall - ranged from .002 to 
.999 with a mean of .29 (SD = .36). The continuous 
score was stratified into 6 balanced subgroups within 
which there were no statistically significant differences 
in the propensity between those with high recall (treat- 
ment group) and those with no/low recall (matched 
control group). There were also no statistically signifi- 
cant differences for any of the 7 variables used in the 
regression within any of the six subgroups (42 tests). 
Results of the propensity score analysis showed that 
71.51% of participants in the treatment group (with high 
recall) practiced academic planning after the MPE com- 
pared to 52.65% of participants in the matched control 
group (with low/no recall), a statistically significant dif- 
ference of 18.60 percentage points (Z test-based P < 
.05). 

Figure 2 presents the simple bar graph comparing the 
results of the analysis between unadjusted and adjusted 
by PSM. The unadjusted difference in APB between the 
treatment group and matched control group was 29.23 
percentage points (71.51% minus 42.28%). The propen- 
sity score results were based on the weighted sum of 
differences in APB among the balanced, statistically 
equivalent subgroups. The adjusted difference between 
the treatment group and matched control group was 
18.6 percentage points (71.51% minus 52.65%). By statis- 
tically balancing the treatment group (high recall) and 
the matched control group (low/no recall) based on the 



propensity scores and all the variables used to construct 
it, the estimate of impact declines 10.63 percentage 
points. The estimated 18.6 percentage point increase in 
APB as an impact of the education can be translated 
into the practice of APB by 670 students in the popula- 
tion (this calculation was based on the total population 
of 3145 undergraduate students). This number of aca- 
demic planners can be attributed to a high recall of the 
education. 

Discussion 

An important feature of a quality education program is 
a regular review of its curricula in order to improve it 
[27]. But, before so doing, teachers and researchers 
would like to see what impact their designed and deliv- 
ered education program has had on changing students' 
learning behavior or achievement. Therefore, this study 
aimed to estimate the impact of teaching a topic in a 
medical education institution in Vietnam on students' 
learning behavior. In particular we sought to quantify 
the effects of the designed and delivered management 
and planning education on students' academic planning 
behavior. We did so using a fairly large sample of such 
students and methods that would help greatly reduce 
selection bias. Because we could not randomly assign 
students to receive or not receive MPE, we used propen- 
sity score matching techniques to contrast the behavior 
of students who did and who did not receive this educa- 
tion but who had been matched on a variety of observed 
background characteristics. We also used some other 
important analyses such as structural equation modeling 
to identify the model of multivariate effect. 

This study provided a combination of tests - logistic 
regression, structural equation modeling and propensity 
score matching - of the effect of the MPE on ideation 
and on APB. In Vietnam, the MPE is delivered for the 
fifth academic year medical students, usually during the 
annual fall (from August to October). Afterwards we 
conducted a data collection. With data from a represen- 
tative sample survey of the students, propensity score 



Table 3 Balance of the propensity scores for MPE exposure: Results from PSM 


Stratum 


Non-Exposed to MPE (A) 


Exposed to MPE (B) 


Total 


Net Difference (B-A) Using "Atts" Command 




N 


N 






1 (range of pscore = 0 - -0.1 )# 


221 


0 


221 




2 (range of pscore = 0.1 - ~0.2)# 


39 


6 


45 




3 (range of pscore = 0.2 - ~0.3)# 


7 


4 


11 




4 (range of pscore = 0.3 - ~0.4)# 


19 


26 


45 


18.6% (p < 0.05) 


5 (range of pscore = 0.4 - ~0.5)# 


7 


20 


27 




6 (range of pscore = 0.5 - 


5 


67 


72 




Total 


298 


123 


421 





Average propensity score = .29; SD = .36; Range = .002 - .999 

# pscore statistics indicate that there is no statistical difference between A and B within each stratum (p > .05), meaning that propensity scores balanced at 6 
strata 
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□ Treatment group 

□ Matched control 
group 

E3 Net difference 



Unadjusted Adjusted by PSM 



Figure 2 Comparison of the unadjusted increase in APB to the increase adjusted by PSM (N = 421). 



matching was able to provide a valid counter-factual 
condition - the matched control group and therefore an 
unbiased estimate of a percentage point increase in aca- 
demic planning behavior that would not have happened 
without the MPE. Specifically, we found that students 
who had a higher recall of MPE contents displayed a 
higher proportion of academic planning behavior com- 
pared to those with a low and no recall (71.51% vs. 
52.65%, p < .05) for a net increase of 18.6 percentage 
points. The MPE reached most of students at the fifth 
and sixth years as many sessions of this topic were com- 
pulsory to students when enrolled in it. An average of 
2.52 MPE contents out of 8 were recalled. Bivariate ana- 
lysis showed that content recall was significantly related 
to ideation and APB, and ideation was also strongly 
related to APB. More important, the multivariate analy- 
sis revealed that the level of ideation and level of MPE 
recall was also related to APB. 

The after-only, cross-sectional regression analysis 
without ideation showed a significant direct effect of 
the communication or education campaign on the 
adoption of a behavior [8,10] after controlling for 
socioeconomic variables. This is as much as many 
studies of mass media impacts are able to do 
[6-8,10,11]. When the composite measure of ideation 
was added to the after-only regression equation, the 
direct effect of the communication and/or education 
was no longer statistically significant because of the 
strong effect of ideation [6-8,10]. However, we found 
in our study that the after-only regression analysis of 
ideation showed that the content recall had the stron- 
gest effect on ideation and the regression analysis of 
APB showed that the recall had the second strongest 
effect on APB after the ideation. Therefore, the 
results confirmed the indirect effects of the education, 
but also its direct effect on behavior, emphasized the 
role of mediating effect of ideation, and provided sup- 
port for the theoretical model that guided the design 
of the education or intervention and the evaluation of 
the results. 



The structural equation modeling could provide sup- 
port for a causal inference. The three equations (repre- 
sented in model 1, Table 2) controlled for potentially 
confounding (socioeconomic) variables that might affect 
behavior. After controlling for these variables, the MPE 
content recall had a significant effect on both ideation 
and APB, and ideation had a significant effect on APB. 
The potential effect of unobserved variables (not in the 
equations) and the reciprocal effect of behavior on idea- 
tion and recall were ruled out by the statistical tests for 
endogeneity. The only criterion missing for a causal 
inference was a counterfactual condition which could 
have been provided only in a controlled experimental 
design. In our study, the counterfactual condition was 
made by propensity score matching in order to create a 
matched controlled group so the comparison of net dif- 
ference would be possible. However, acceptance of such 
a causal inference for MPE and ideation on APB does 
not necessarily mean that other causes were not also 
operating. Students would approach or may be exposed 
to other sources of MPE such as internet, library or 
others. But at least in this study we could argue that the 
effect on APB would be as a result of the MPE per se 
which was designed and delivered by the Department of 
Health Organization and Management because we mea- 
sured the recall of the key contents taught only at the 
university. Comparing between the treatment group and 
matched control group, a 18.6 percentage point increase 
in APB after the education delivered may sound small. 
However, because the sample of 421 represents a popu- 
lation of 3,145 students, the actual net increase in the 
number of students practicing APB is estimated to be 
670. 

Although many efforts have been made, our study can 
have several limitations. First, because many sessions of 
MPE - lectures and practicum - were compulsory, there 
would be an efficient way to reach students as well as 
MPE. In this situation, the risk of falsely rejecting an 
effective treatment (Type II error) appears to be greater 
than the risk of falsely rejecting the null hypothesis 
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(Type I error) [28]. Further, due to the nature of self- 
reported research design, recall bias would be inherent. 
Some students, perhaps because of their self-esteem, 
could over- or under-estimate their behavioral 
responses. However, as this study designed a survey on 
a fairly large, representative sample with anonymous 
and confidential commitment, it would partly reduce 
such a bias. Also, since students self-reported their aca- 
demic planning behavior, further studies should com- 
bine self-reporting of APB and observation or collection 
of actual plans (study notes, schedules, timetable, etc) of 
students. Moreover, as a cross-sectional study, this 
design may preclude the order of causality; therefore, a 
longitudinal study is needed to address this concern. 

Interpretations of this study provide a theoretical as 
well as a practical implication. Consistent with the lit- 
erature, our study informed the indirect effect of the 
education or communication (message recall) on beha- 
vior or the intervening role of ideation between educa- 
tion and behavior in the theoretical framework. At the 
same time, we also indicated the direct effect of the edu- 
cation on behavior. This suggests that designs of educa- 
tion evaluation should include ideation in addition to 
the education or recall in order to obtain a holistic theo- 
retical model to support research. Another practical 
concern is: Should lecturers or institution leaders take 
action based on this conclusion? Should they review 
curricula and teaching using structural equation model- 
ing and propensity score matching? The results of this 
study suggest that such a technique, a reliable, but a still 
neglected method of research and evaluation in many 
educational contexts, should be rolled out to other 
topics of education evaluation. 

Conclusion 

Adopting PSM has made it possible to create a matched 
control group, allowing for estimating the net impact of 
managerial education on APB. 

The results confirmed the indirect effect of the educa- 
tion on behavior or the intervening role of ideation 
between education and behavior. The direct effect of the 
education on behavior was also identified. 

PSM adjusting for the possible confounders, there is a 
statistically significant difference in APB between stu- 
dents who did and did not recall management contents. 

The study provided a theoretical as well as a practical 
implication to guide the design of the education and the 
evaluation of teaching and/or curriculum 
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